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ABSTRACT 



The evaluation of a Goals 2000 initiative in New York state 
is explored through these four dimensions: focus, innovation, involvement of 
teachers in program evaluation, and the relevance of program evaluation to 
educational practice. In central southern New York state, seven public and 
six nonpublic schools are collaborators in the Goals 2000 professional 
development initiative studied. Although evaluation data came from many 
sources, the primary method of data collection, and the focus of this study, 
was a process known as a "Collaborative Review of Student Work" (CRSW) . CRSW 
is a process that brings peer coaches and coached teachers together to 
examine samples of student work carefully, giving teachers a forum to examine 
the connection between their practice and student performance. Evaluation 
shows the CRSW to be an evaluation tool that makes contributions to 
educational research and program evaluation along the four dimensions 
identified. Some technical considerations in using CRSW are outlined. 
(Contains 5 figures and 15 references.) (SLD) 
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INTRODUCTION 



Attempts to evaluate Goals 2000 initiatives in New York State have fallen victim 
to criticism from two opposing camps. In one camp are the educational reformers who 
call for program evaluation that moves beyond traditional evaluation models~pre/post 
test scores or participants as information givers— to create forums for evaluation that 
support teachers as intellectuals. In this model of evaluation, teachers are creating habits 
of inquiry, carrying on structured dialogue that creates new knowledge about pedagogy 
and policy. ’ 

On the other side of the debate are those reformers who argue that happy, 
involved teachers who know how to think are not necessarily indicators of school 
improvement. They suggest that claiming success when there has been no demonstration 
of improved student achievement is like ignoring the fact that the emperor has no clothes. 
These advocates of a more conventional evaluation model suggest that evaluation of 
educational initiatives must focus on the “bottom line”-student achievement. In this 
evaluation model, success in change initiatives can only be claimed when objective, 
valid, and quantifiable data reveal improved student achievement^ 

This ideological divergence raises an opportunity for evaluators and researchers. 

Is there a middle ground that utilizes the best thinking of both ideological camps? How 
can evaluators of school change initiatives support the development of habits of inquiry 
and at the same time use meaningful student performance data? This paper presents an 
educational research design that places student work at the center of a participatory 
qualitative program evaluation. Specifically, the paper explores the evaluation of a Goals 
2000 initiative along four domains: Focus on student work, innovation in the 
examination of student work, substantive involvement of teachers in program evaluation, 
and the relevance of program evaluation to educational practice. 

BACKGROUND OF THE PROJECT 

Since 1994, New York State has received federal funds as part of the Goals 2000: 
Educate America Act. The Act prompted the appropriation of federal funds to “support 
initiatives at the local level that build the capacity of schools/districts to enable all 
students to achieve the new State Learning Standards.”^ 



' From Changing Minds, “Teachers looking closely at students and their work.” Spring 1998. Bulletin 
#13, Michigan State University. 

^ HofF, D. (1999). Falling Short. Education Week v. 10 (5). Pp.10-1 1. See also: Howerter, D. (1996). 
Dumb and Dumber: Goals 2000 and QBE. http:www.otherside.net/goal2000.htm. 

^Kadamus, J. (1997). Goals 2000: Educate America Act Request for Proposals. The University of the 
State of New York. The State Education Department. 
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In central southern New York State, seven public and six nonpublic school 
districts are collaborators in a one year Goals 2000 Professional Development (PD) 
initiative designed to achieve three goals: 

• Infuse literacy instruction (reading, writing, speaking, and listening) within 
and across the districts English language arts, math, science, and technology 
curriculum; 

• Incorporate real-world, problem based learning into an integrated math, 
science, and technology curriculum; and 

• Integrate the use of technology as an instructional tool into the English 
language arts, math, science, and technology curriculum. 

Toward these ends, the initiative employs three professional development strategies: 
Professional training, teacher peer coaching, and electronic linkages to learning. Figure 1 
outlines the scope and sequence of training activities. 

The initiative was developed by a consortium of the participating schools and 
resource providers such as Board of Cooperative Educational Services (BOCES), the 
Economic Opportunity Council, the Coimty Chamber of Commerce, institutions of higher 
education, and a number of community based organizations. The planning group, known 
as the Southern Tier Consortium, has stated several core beliefs and responsibilities that 
drive the project design. Among the stated beliefs which appear in the project plan: 

• The purpose of staff development is to impact students learning in the classroom. 

• All players (learners, districts, sponsors, consultants) must be willing to alter 
systemic barriers to quality learning 

• All players must be held accoimtable for and feel a sense of ownership for 
success. 

These core principles have influenced the design of both the PD initiative and the 
evaluation. Every attempt has been made to create an evaluation that responds to the 
stated values of the project planners. 

THE PROBLEM 

With the sim about to set on the Goals 2000: Educate America Act, proponents of 
the initiative are scrambling to offer defensible evidence of project success. Acting 
Deputy Secretary of Education Marshall S. Smith declared in a 1998 interview, "The 
measure of Goals 2000 is that we've got 47 or 48 states with standards. A lot of states are 
on the way toward aligned assessments. We've begim to move the field in a way that 
they are grappling with performance standards."^ The criticism endures that Goals 2000 



D. (1998). Controversial Goals 2000 to face new uncertainties. Education Week. Jan 21. pp 26-28. 



monies have been directed at initiatives that are unrelated or indirectly related to 
standards and assessment. There is, to date, scant evidence that Goals 2000 professional 
development initiatives have improved student performance or the manner in which 
students are assessed. It has been difficult, if not impossible, to isolate the impact of 
Goals 2000 initiatives on student performance.^ 

Several difficulties have plagued Goals 2000 evaluation attempts. Given the 
complexity of the change initiatives, which occur in an already complex educational 
system, it has been extremely difficult to get 'clean' data — ^that is, credible data that 
demonstrates a traceable connection between the input (Goals 2000 professional 
development initiative) and the output (student performance). A sampling of the enduring 
troubles which have plagued attempts to evaluate initiatives are outlined in the section 
which follows. 

1. The mismatch of measures : In attempts to show the impact of Goals 2000 
initiatives on student learning, there has been a tendency to mismatch measures of student 
performance with claims that Goals 2000 is or is not effective. Most often, standardized 
test scores are publicly reported as evidence that a particular initiative has lead to (or 
caused) an increase in student test scores. In New York State, the PEP tests and Regents 
examination scores are watched carefully as indicators of student performance and a 
school's progress. When these scores drop, as they are likely to do at the beginning of a 
change initiative, education critics are quick to declare that a particular initiative (e.g.: 
the adoption of a new reading program) has caused the drop in scores. 

In the case of Goals 2000 initiatives, standardized tests may be particularly unfit, 
for they measure overall performance in a particular subject by a selected grade level--the 
entire third grade, for example. In the following year, the scores of a new third grade 
cohort are reported. These tests are not designed to capture incremental change in 
student performance or change in teacher behavior. Nor do these measures give teachers 
any useful feedback on their practice. We are not suggesting that over an extended 
period of time it is unreasonable to suggest that school-wide test scores should increase, 
but it’s misleading, if not unethical, to claim that changes in a particular test score are 
related to an isolated Goals 2000 project. 

2. Tangled Variables : As Goals 2000 initiatives are carried out in schools, any 
number of changes are occurring in parallel. In the schools in this study, there are 
changes in staff, fluctuations in student enrollment, programmatic changes and the like 
which will occur throughout the year of the project. The fallout of any one project, 
initiative, or systemic change that occurs throughout a school year resembles an M.C. 
Escher drawing, where it's impossible to see the end of one element, the beginning of the 
next, or the relationship between the elements. This makes claims of correlation 
particularly difficult. How can we claim that changes in overall student performance are 
correlated with a particular new initiative? How can we isolate, or hold constant. 



’ Hayford, P. (1998). New York State Department of Education. Office of Innovative Programs. Phone 
conversation. 
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particular variables to demonstrate a correlation between student performance and a 
certain initiative? 

3. A protection of professional precepts . Teachers are necessarily skeptical of 
“opportunities” for involvement in evaluation projects which turn them into arbiters of 
each others’ practice. In a 1995 study, Joyce and Showers found that “When teachers try 
to give one another feedback, collaborative activity tends to disintegrate. Peer coaches 
[tend to] slip into “supervisory, evaluative comments.”® This is not the intent of peer 
coaching, and teachers are quick to identify inappropriate uses of the coaching process. 
Even in the short period that the previously described New York initiative has been 
underway, we have observed a careful and cautious attention to the use of evaluation 
tools. The participants have asked challenging and justifiable questions about how data 
will be used and have resisted any evaluation method which could potentially place 
teachers in the position of evaluating each other. It is possible that the fear of ‘coaching 
turned into evaluation’ inhibits teachers’ ability to give candid insights about their own or 
others’ work. 

4. Vague language : Grant proposals, sometimes written by paid grant writers 
who may not understand educational jargon, are full of provocative language included to 
catch the eye of funders. Thus, proposals are submitted which promise to do everything 
except leap tall buildings in a single bound. Stated objectives tend to be long on 
captivating language and short on observable, measurable goals. Are Goals 2000 local 
initiatives doing what they proposed to do when they responded to the State's RFP? It's 
difficult to know, when trying to measure slippery concepts such as "the infusion of 
English language arts," or the "integration of technology. . . " or "capacity building at the 
district level. " 

EVALUATION DESIGN 

The evaluation described in this paper was designed to respond to the stated 
beliefs of the consortium and to the aforementioned difficulties associated with 
evaluation. Figure 2 offers a graphic representation of the overall evaluation design. 

Data are drawn from peer coaches’ classroom observations of coached teachers, periodic 
focus group meetings with the peer coaches, and reflective writing by the teachers 
themselves. For the purpose of this paper, however, we will focus on a primary method 
of data collection, a process known as a “Collaborative Review of Student Work 
(CRSW).” 

The Collaborative Review of Student Work is a process that brings peer coaches 
and coached teachers together to carefully examine samples of student work. The work 
is brought to the conference by one or more of the participating teachers. The goal of the 
CRSW conference is to provide a forum for teachers to examine the connection between 



Showers, B. andJoyce, B. (1996). The evolution of peer coaching. Educational Leadership 53 (6). P.15. 
See also: Showers, B. and Joyce, B. (1995)Student achievement through staff development: 

Fundamentals of school renewal. 2“‘'Ed. White Plains, NY: Longman. 
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their practice and student performance, and to encourage a critical and comprehensive 
look at student work. 

The Collaborative Review of Student Work (CRSW) 

The Collaborative Review of Student Work takes as its framework the design of 
the Collaborative Assessment Conference, developed in 1988 by Steven Seidel and others 
at Harvard’s Project Zero. The Collaborative Assessment Conference lasts between 45 
minutes and an hour, and it is designed to encourage educators to look at student work, 
describe student work, and listen to others describe student work — all to promote 
“wondering,” or thinking deeply, about the work. Figure 3 offers a graphic presentation 
of this interaction. The Collaborative Assessment Conference includes the following 
basic steps. 

1. Getting started; 

2. Describing the work; 

3. Raising questions; 

4. Speculating about what the student is working on; 

5. TTie “presenting teacher” speaks; 

6. Implications for teaching and learning; and 

7. Final reflections.’ 

First, the conference gets started by selecting a facilitator and handing out the 
student work to be discussed. The presenting teacher does not offer any comments about 
the work or the assignment. Then foe group describes any aspect of foe work they notice, 
without making judgments about quality. Following foe descriptive phase, foe presenting 
teacher takes notes while foe group asks questions about foe student, foe assignment, foe 
cumculum, etc., but does not respond to foe questions at this time. The group then 
speculates about what foe student is working on — such as how foe student was trying to 
complete foe assignment, skills foe student might be trying to master, or ideas she or he 
was trying to express. At foe end of this “guessing period,” foe presenting teacher 
responds to foe earlier questions, and adds her perspective on each of foe previous phases 
of foe conference. At this point, everyone is invited to share ideas about foe implications 
of foe examination of work far for teaching and learning. Finally, participants have a 
chance to reflect on foe process of their own thoughts throughout foe conference. 

There are five basic principles that underlie foe process. These same principles 
apply to foe Collaborative Review of Student Work: 

1 . Start by looking at foe work knowing little or nothing about foe context in 
which foe work was produced. 

2. Describe foe piece, keeping in mind that no observation is too obvious. 
Withhold judgments as much as possible. 

3. Most students approach their work with purpose and intent even if it is not 
immediately apparent. 




’ Cuslunan, K. (1996). Looking collaboratively at student work: An essential toolkit. Horace, The 
Coalition of Essential Schools, 13(2), p. 3. 
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4. When speculating about what the student was trying to accomplish, be ready 
to cite examples from the work. Keep coming back to the work. 

5. Don’t worry if you come away with more questions than answers.* 



The process for Collaborative Review of Student Work (CRSW) draws heavily 
upon the steps of the Collaborative Assessment Conference and these five principles. 

The CRSW conference is specific in regard to the roles of conference participants, the 
nature of student work, and the process of the conference itself. 

Roles 

The peer coach will serve as facilitator to keep reviewers focused on the student 
work and monitor time. The facilitator must also protect presenters from 
evaluative/judgmental statements by reframing or redirecting inappropriate comments or 
questions from reviewers and seek and support thoughtful discussion. 

One or two presenting teachers will bring a piece of student work to the 
conference. This may be written work, project work, and audio- or videotape — any 
medium is appropriate as long as each CRSW participant is able to easily see or hear the 
work. The student work should be anonymous, current, and free of teacher comments or 
grades. Four to six reviewer/participants offer insights and raise questions about the 
student work. 

Process 

Getting Started (10 minutes); All participants introduce themselves, and the 
facilitator offers an overview of the CRSW process. This overview is particularly 
important when participants are not familiar with the process. The presenting teacher 
distributes work or displays it where all participants can see or hear it, but the teacher 
does not comment on the work. Participants observe the work and write down notes or 
questions for their own use. 

Descriptive Phase (15 minutes): Participants describe what they see in the work 
as non-judgmentally as possible. The following questions are offered by the facilitator to 
guide the discussion; 

• What do you see in this student’s work that is surprising or interesting? 

• What do you learn about how this student thinks and learns? 

• What skills do you see exhibited in this work? Extension: Which literacy 
skills or standards-based skills do you find in this work? 

• What tools did the student use to organize the work (e.g.; technology, 
graphics) 

• What were the apparent intended outcomes of this student work? 



From Discussing student work gives teachers a new perspective: Project Zero’s “Rounds” adapts a 
medical tradition for teachers. The Harvard Education Letter, July/August, 1998, p. 8. 



Question phase (15 minutes): Participants ask questions of the presenting teacher. 
These can be as simple as clarifying questions about grade level/content area, or they 
may be more complex questions about the link between what was done in the classroom 
and what is being presented. The facilitator may choose to guide the question phase by 
asking “What questions about teaching and assessment does looking at this student’s 
work raise for you?” or “What questions would you like to pursue further?” 

Presenting teacher responds (10 minutes): The presenting teacher responds to 
questions raised by participants, talks about the intended outcomes, and the assignment in 
relation to what has actually been produced. The teacher may make connections between 
practice and outcome. Finally, the presenting teacher reflects on his or her own learning 
as a result of the collaborative review. 

Debrief stage (10 minutes): All participate, with the facilitator guiding the 
discussion. The main questions for debriefing include “What did you learn from listening 
to your colleagues that was interesting or surprising?” and “Are there ideas or strategies 
that you would like to try in your own classroom?” Figure 4 depicts the protocol for 
facilitators. 

Upon completion of the CRSW, peer coaches are asked to complete a post-CRSW 
conference observation. The form used by peer coaches is presented in Figure 5. 

Upon completion of each round of Collaborative Reviews, the evaluation team meets 
with peer coaches for a focus group discussion which examines the process itself. 

SIGNIFICANCE OF THE DESIGN 

We believe that the Collaborative Review of Student Work as an evaluation tool 
makes contributions to educational research and program evaluation along four 
dimensions. 

1. Focus on Student Work . Students, and in particular students’ work, have 
historically been left out of Goals 2000 program evaluations. In the design described 
here, the focus of the evaluation is clear: We want to know what students are doing in 
the classroom specific to the goals of the project. From the CRSW conference, we hope 
to help teachers understand how their own practice influences student performance. It is, 
however, what teachers are seeing in the student work that drives any discussion of 
practice. 

2. Iimovation in the examination of student work . The Collaborative Review of 
Student Work conference is grounded in the work of Harvard’s Project Zero. Influenced 
by Howard Gardener’s with the Prospect School in Vermont, Project Zero has developed, 
tested, and revised an approach to collaborative assessment which brings teachers 
together to examine the impact of their practice on the student learning and outcomes. 
Under the direction of Steve Seidel and others. Project Zero has developed a model that 
helps teachers analyze student work, to promote change in their classroom practice, and 
to answer the question “What are we doing (or what do we need to do) here to get student 
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work to a higher level?”^ Project Zero’s significant contribution is grounded in a 
research base that is giving teachers better clues as to how to move students to higher 
levels of learning. The process has received acclaim as a professional development tool 
“centered on discussions of standards and student work. . .” and that helps teachers in low 
performing districts “move kids off the bottom.” The Project Zero model, and others 
like it, have not yet been utilized as a component of a program evaluation in schools. Our 
adaptation of the model may serve as a pilot test of the model’s utility and adequacy as an 
evaluation tool for program evaluation. 

3. Substantive involvement of teachers in program evaluation. Drawing from the 
work of Guba and Lincoln (1989), Patton (1990) and others, the design is committed to 
the involvement of the stakeholders at every stage of the evaluation process. A body of 
research on participatory evaluation suggests that the meaningful involvement of 
educational personnel in all stages of the evaluation process will increase the likelihood 
that teachers will use the evaluation results — ^taking new action and adapting their actions 
based on the findings from the evaluation. 

Teachers in this project have been involved in every stage of the evaluation. In 
the preparatory stages, teachers and peer coaches worked with the evaluation team to 
develop observable indicators of progress toward the project’s goals: Infusing Literacy, 
incorporating real world, problem-based learning, and integrating technology. Working 
in groups, teachers answered the question “what would it look like if this goal were 
achieved in terms of student work, teacher instruction, and/or classroom activity? These 
indicators have become lenses through which evaluators view data collected during 
CRSW conferences. 

Perhaps the greatest value for teachers imdertaking collaborative reviews of 
student work is that they are using a tool that is both central to the program evaluation 
and ideally useful in their practice beyond this project. Evidence of the utility of this 
process for teachers can be foimd in reflective writing gathered at the conclusion of the 
first roimd of collaborative conferences (November, 1998): 

“In this peer review session, myself and a few other teachers sat down and looked 
at two pieces of writing...! learned from the session how to look at a student ’s 
work more objectively. The session served as a training in which we really 
analyzed a work, but did not place judgments on the work I gathered from my 
colleagues many ideas to try in my own classroom. ” 

“It is interesting to hear other people ’s thoughts on something that I am close to 
and may not have been able to see. ” 

“There are many implications for teaching and learning in this process. It was 
interesting to see the different perspectives of other teachers ...they had insights 
that I did not have. ” 



® Lewis, A. (1998). Teachers in the driver’s seat. The Harvard Education Letter. XIV (2). March/April. 

Resnick, L. (1997). Address to the Center for Research on Evaluation, Standards, and Student Testing 
conference. Cited in Lewis. 
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“I learned that although we are unique in terms of the discipline that we teach, 

we have many commonalties, and that in order to be effective educators we must 

listen to and learn from our colleagues. ” 

Data from teachers in this project are not limited to survey responses or interview d at a 
Teachers are looking at the work of their students and making cormections to their 
practice. 

4. Relevance of program evaluation to educational practice . In A New Vision for 
Professional Development, Sparks and Hirsch (1997) call for professional development 
for educators that is linked to a larger vision for school change.** Based on an 
organizational change perspective, Sparks and Hirsch argue that teachers must see 
cormections between their professional development and their every day practice. In 
addition, they need to be involved in carving out a long term agenda for change that is 
based on a model of continuous professional improvement. The model described in this 
paper leads teachers through a series of ongoing, reflective conferences in which they 
closely examine student work and identify areas for change in their school and their 
practice. The model promotes evaluation in support of institutional change. 

TECHNICAL CONSIDERATIONS 

The use of CRS W as an evaluation tool in a large, multi-district project presented 
a number of technical considerations for the evaluation of the Goals 2000 initiative. 

Role of peer coaches Perhaps first and foremost, the training and utilization of 
peer coaches as the primary gatherers of data for the project presents a significant 
concern. The peer coaches in this project were not trained in evaluation methods, nor 
were they trained in the CRSW protocol prior to the start of the initiative. In fact, none of 
the peer coaches were familiar with Harvard’s Collaborative Assessment Conference 
prior to the start of the evaluation project, so the primary data collection method required 
rapid training of peer coaches. 

In addition to providing several brief articles describing the work of Seidel and 
others at Harvard’s Project Zero, a structured session was offered, during which the 
project evaluators described the process, then lead peer coaches through a role play of the 
process (using student work “donated” by two peer coaches). Peer coaches practiced 
their roles as facilitators and became familiar with the roles they would ask teachers to 
play as presenting teachers and participants. The focus of the session was on 
faimliarizing peer coaches with the protocol, and sensitizing them to the possibility that 
the discussion could evolve from a focus on student work to an “advice session” or even 
less appropriate for the evaluation, an evaluation of the presenting teacher’s skills. 

Unfortunately, not all of the peer coaches were present for the tr aining session. 



*' Sparks, D. and Hirsh, S. (1997). A New Vision for Professional Development. Association for 
Supervision and Curriculum Development. Alexandria, VA. 
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As a result, it is difficult to assess the degree of familiarity with the process and the 
attendant protocol among those peer coaches who were trained by their colleagues rather 
than by the evaluation team. Just as information disseminated time and again is subject 
to distortion, there is a concern that not all peer coaches were adequately trained to 
conduct CRSW conferences in a consistent marmer. 

In addition, there is evidence from evaluator observation of CRSW conferences 
that some individualization of the process has occurred even among those peer coaches 
who were initially trained in the process. As a result, the CRSW conferences vary as a 
result of unequal training and as a result of peer coach individualization, either as a result 
of the peer coach’s interpersonal style or as an outgrowth of a need or desire expressed by 
participants. For example, one peer coach truncated the crucial final step of the process 
in two CRSW conferences because teachers could not stay long enough to complete the 
entire process (observation, March 1999). Although the teachers participating in the 
conference seemed receptive to the idea of meeting to see what was happening in other 
classrooms, they did not get the advantage of reflecting on the questions raised, nor did 
they reflect on the process itself. As a result, their experience of CRSW was more similar 
to “show and tell” than to the ideas present in the Collaborative Assessment Conference 
on which the CRSW process is based. Clearly, this presents a technical concern with 
regard to the use of CRSW as an evaluation tool. 

Teacher hesitation/buv-in A second serious technical consideration is the degree 
to which teachers are willing to “buy in” to the idea of CRSW, especially as part of a 
formal evaluation. Even among teachers who are willing to consider participating in 
CRSW, the limited time available for such “extras” during the school day, and even 
afterward, presents a concern. Peer coaches find it hard to schedule an hour-long CRSW 
conference. As one peer coach put it, “It is difficult for me to sell ‘reflecting on student 
work’ to teachers with a 25-minute prep period during the day. Their after-school time is 
essential for planning and prep” (Focus group, November, 1998). Clearly, time 
constraints present a challenge for the use of this process in engaging teachers in 
evaluation. 

The nature of discussion Using CRSW as an evaluation tool relies heavily on 
capturing the nature of discussions from which the members of the evaluation team are 
routinely absent. How well participants are able to recreate important aspects of the 
actual CRSW discussions is a concern for the evaluation process. A number of protocols 
are designed to ensure that crucial information reaches the evaluators including ftie post 
conference observation form found in Appendix 5 and the teachers’ reflective writing 
which follows the CRSW. The possibility that important data is not captured exists, 
however. One peer coach noted, “Collaborative review — if the purpose is to capture our 
conversation, how might I better do so? Most of the good stuff didn’t make it to paper” 
(Focus group, November, 1998). The degree to which participants are able to furnish 
adequate information about the CRSW process has a direct impact on the utility of the 
process as an evaluation tool. 
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District commitment Finally, given the evaluation design, it is vital to consider 
the level of commitment on the part of districts participating in the evaluation. Sufficient 
time for peer coaches to facilitate CRSW, and for teachers to participate are needed. 

Some peer coaches are stretched to the point of breaking. As the harried peer coach 
explains, “A constraint: being in two districts/six buildings.... I fear for my credibility 
because I can’t always follow through” (Focus group, November 1998). One district 
provided only ten weeks’ classroom leave for teachers functioning as peer coaches. The 
Goals 2000 project was implemented in such a truncated manner in this school that the 
CRSW process was virtually useless. A peer coach from this district notes, “[The] ten- 
week plan has limited my ability to gather teachers to focus on student work. I would 
need a better district ‘buy-in’ which might afford some 30 minute release time events for 
classroom teachers.” (Focus group, November, 1998). 

STRENGTHS AND WEAKNESSES OF THE CRSW PROCESS AS AN 
EVALUATION TOOL 

Despite the technical considerations that suggest that getting even one CRSW off 
the ground would be tantamount to a miracle, to date, approximately 20 CRSW 
conferences have been conducted. The data supplied by the participants, and by evaluator 
observations of conferences suggests that there are strengths and weaknesses involved in 
using CRSW as an evaluation tool. 

Focus on student work A clear strength of the CRSW process lies in its ability to 
focus the attention of coached teachers, peer coaches, and evaluators on the bottom line: 
student work. A comment from the first roimd of teacher reflections (November 1998) 
indicates the focus on student work: 

“Assessing a piece together with my team was helpfiil in that I became acquainted 
with the expectations and reaction of teachers working at my grade level. We 
talked about what was good, what needed improvement, and what reflected 
student thinking. We also talked about how the presentation with the assignment 
could’ve affected the quality of the piece. ” 

A potential weakness in the process involves the question of which student work 
comes to the conference table. While CRSW is designed to encourage teachers to bring a 
broad range of student work — ^from the superlative to the problematic — ^to the conference 
table, it is difficult to discern whether the work being discussed is an accurate 
representation of the range of student work being done in the classroom. It is possible 
that teachers in the early stages of the evaluation may be most comfortable offering high- 
quality exemplars of student work. For example, in a recent CRSW observation (March 
1999), the presenting teacher stressed the fact that the student work presented received all 
the points possible for the project (500 points). It is easy to imderstand why teachers 
might select shining examples to share with their colleagues, particularly given limited 
familiarity with the CRSW process and the linkage of the conferences to “evaluation.” 

On the other hand, teachers may view the CRSW process as an opportunity to bring 
“problem work” in the hope of gleaning worthwhile suggestions. The problem for 
evaluation, however, is that the types of work presented strongly influence the findings of 
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the evaluation process: If the student work actually reaching the conference table does 
not reflect the range of work being done, the evaluation potentially misses the 
opportunity to identify strengths and weaknesses of the initiative. 

An additional weakness arises from the fact that the CRSW process in each 
school has not been limited to an examination of one presenting teacher or student over 
the course of the evaluation. Although inviting a different teacher to present work at each 
of the three data collection points encourages participation, it eliminates the possibility of 
making longitudinal comparisons of the work assigned. 

Innovation in examining student work In addition to placing the focus squarely 
on student work, the CRSW process also facilitates innovation in thinking about student 
work. One particular strength of this innovation is cross-curricular examination of work. 
CRSW participants note the interdisciplinary advantage, not just for themselves, but for 
their students: 

“In twenty years of teaching, this is the first time I’ve had the opportunity to share 
my kids ’ work with teachers outside my classroom. I’m the only psychology 
teacher, so where was the opportunity to look collaboratively at what my students 
are able to do? Now there are people who teach English giving feedback on 
whether [students] are writing well in my class. ’’ 

“I found this activity and discussion fascinating — all of the different subjects 
could be pulled in to strengthen the whole activity, focusing on student learning. ’’ 

“I found this interesting because I was able to hear how other teachers from 
different subject areas examined the student ’s work We all came up with a way 
to use the assignment in our classrooms. ...After our discussion, it was clear that 
as a teacher if we all work together as a team with more projects that include all 
the courses, the student will benefit the most. ’’ 

The interdisciplinary nature of CRSW can present potential for resistance, insofar 
as all teachers are not equally competent or comfortable in all subject areas. This seems 
to be the case with the goal of infusing literacy, for example. In a CRSW observed by an 
evaluator, a science teacher protested that she was so unfamiliar with a student paper of 
the sort being considered that she could not even read it. The work being discussed was a 
story written by a student with serious deficits in the mechanics of writing, such as 
punctuation, capitalization, and spelling. From the science teacher’s perspective, analysis 
of this work was clearly out of her province. In a teacher reflection written at the 
conclusion of another CRSW, the peer coach reported that a math teacher “didn’t feel 
knowledgeable enough about grammar to grade for it in math class.” The peer coach 
suggested that “quality writing seemed important to all teachers, but they were a little 
shaky on grading for it or expecting it in their classes.” This suggests that as an 
evaluation tool, the CRSW hasn’t lead to a full recognition of the range of student 
performance, as participating teachers tend to self-censor their comments based on their 
perceived competencies. 
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Teacher involvement in evaluation There is evidence that the CRSW process 
directly involves teachers in the evaluation process. CRSW underscores the goals of the 
Goals 2000 initiative, and reinforces teachers’ awareness of those goals and their ability 
to contribute data toward evaluation. Teachers in each school have been involved in 
CRSW conferences, which provide a critical source of data for the Goals 2000 
evaluation. These teachers are most aware of the potential for success or failure of the 
project being evaluated, and the CRSW provides a gauge for attainment of project goals. 
As an evaluation tool, the CRSW seems to provide data that is both generated and 
utilized by those closest to the initiative. 

A weakness, however, arises when we examine the number of teachers involved. 
Because of the time constraints on the average classroom teacher, it can be difficult to 
schedule CRSW conferences for more than five or six participants. As a result, a large 
population of teachers do not participate in CRSW, and are still largely uninvolved in the 
evaluation process. While CRSW does encourage useful participation on the part of 
teachers who are willing and able to devote an hour to the process, the voluntary CRSW 
model offers no advantage over non-participatory designs for those teachers who elect 
not to participate. 

Relevance to educational practice The fact that some teachers have participated 
in more than one CRSW, which are usually scheduled during teachers’ limited free time 
suggests that teachers find the process relevant to their educational practice. Teacher 
feedback from the CRSW process suggests that teachers have developed questions — if 
not answers — ^regarding practice oriented issues: 

“In listening to discussions about this [work], I have realized how difficult it is to 
adapt a test to higher level thinking skills...! learned that intended student 
outcomes must be clear before you can develop an assessment. ” 

“When I do this activity again, I would talk to the English teacher about what 
activities I could do with the students to help them assess their writing. I would 
talk to the science teacher about using more descriptive words. ” 

“I learned I would rather like to incorporate more model and guided practice 
work into my own teaching. I discovered I tend to focus in on one topic and 
process rather than specifics of writing. ” 

“I picked up a few new ideas and some things from different viewpoints than I 
had before this discussion began. I will be rewriting assignments to include the 
variety of means for students to demonstrate knowledge. I will consider changing 
the way I handle spelling errors on students ’ work ” 

The primary weakness of the process in terms of its relevance to educational 
practice is the possibility that peer coaches and CRSW participants may fail to make the 
vital link between the questions raised during the conference and their applicability to 
actual practice in other classrooms, or with other students. One example of this surfaced 
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during a recent conference (3/99) when a presenting teacher reported, “I tell my students 
that these are science reports. They’re supposed to be dry and boring. The students want 
to write like they do in English class.” The teacher went on to say, “There is some 
integration here, but I couldn’t really link this to the English standards.” The teacher 
failed to make an explicit connection between his assignment and one of the NYS 
English Language Arts Standards, which expects students to “. . .write to acquire and 
transmit information.” The presenting teacher clearly viewed his work as limited to 
science. This illustrates a limitation of the tool — if teachers don’t have an “a-ha” 
moment during the conference, nothing in the process provides for an ‘intervention’ that 
directly connects teacher reflection to classroom practice. 

QUESTIONS FOR FURTHER RESEARCH 

Significant questions remain to be answered regarding the role of CRSW in 
facilitating a new model of evaluation. Issues of focus, participation and inclusion, and 
systemic barriers all present opportunities for fiirther research. 

While the focus of CRSW is ostensibly student work, questions arise as the result 
of the intimate connections between teaching and learning. At times it is unclear whether 
the process can evaluate student work 'without also commenting — at least obliquely — on 
teacher practice. As one teacher noted in the teacher reflections, “I feel that having my 
student’s work made public makes me feel more self-conscious about my own teaching 
abilities. It is almost a reflection of me and my abilities as an effective teacher” 
(November, 1 998). Given the hesitance of many teachers to review other teachers’ 
practice, this may limit the degree to which different or disparate ideas are likely to result 
from a CRSW conference. This perception may also stifle the ability of CRSW 
participants to disagree about the work being discussed, encouraging them to focus only 
on positive aspects to avoid the appearance of criticizing a colleague. Further research is 
needed to determine if this is a common perception, and if so, the implications for the use 
of CRSW as an evaluation tool. 

One of the most severe limitations of the process is related to sampling: if only 
the most engaged teachers participate, and they tend to bring work that is not 
representative of a wide range of student performance, how can CRSW provide a realistic 
evaluation of the impact of staff development on student work in general? Further 
research is necessary to identify ways to increase both numbers and diversity of CRSW 
participants and student work that is being discussed if CRSW is to contribute 
meaningfully to educational evaluation. 

Finally, further research must examine the systemic barriers to the utilization of 
CRSW or other interdisciplinary methods of evaluation. In the focus group discussion, 
peer coaches reflected the resistance present in many systems: 

"I know what I wish is that we would have a different structure in school... where 
we could teach interdisciplinary units rather than having a fragmented, 
incomplete team concept. ” 
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“We have the so-called four subjects as a team, but every teacher still plans 
individually. ” 

As long as schools as institutions fail to offer the necessary flexibility and 
resources for teachers to examine and evaluate their own work via CRSW or other 
methods, and then empower teachers to implement the ideas and concepts they gain from 
these processes, the emperor is very likely to remain underdressed. 
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FIGURE 1: SCOPE AND SEQUENCE OF TRAINING ACTIVITIES 
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FIGURE 2: EVALUATION DESIGN 




‘Cornell evaluation team will develop and teach protocol 
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FIGURE 3: EFFECT OF COLLABORATIVE REVIEW 




Adapted from “Teachers looking closely at students and their work,” Changing 
Minds, Bulletin 13, Spring 1998, Michigan State University, P. 4. 
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FIGURE 4: FACILITATOR’S PROTOCOL 



The Collaborative Review of Student Work (CRSW) Conference 
Protocol for Facilitators 



Goals of the CRSW conference: 



provide a forum for teachers to examine the connection between their practice and 
student performance. 

encourage a critical and comprehensive look at student work. 

Roles 



The Facilitator (1): 

keeps the reviewers focused on student work 
monitors time and adjusts time as needed 

protects presenters by reframing or redirecting inappropriate comments or 
questions from reviewers 
seeks and supports thoughtful discussion 



One or Two Presenting Teacher(s) 

bring a piece of student work to the conference. This may be written work, 
project work, an audio or videotape — any medium is appropriate. 

Work should be anonymous, current, and free of any comments or grades. 



Four to six reviewer/participants: 

offer insights and raise questions, guided by the protocol 



Process 



10’: GETTING STARTED 



Everyone introduces themselves 
Facilitator reviews the process 

Presenting teacher distributes student work or displays it so all can see it. Teacher 
doesn’t offer any information about the work. 

Participants observe the work and write down notes or questions for their Own 
use. 
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15’: ANALYSIS PHASE 



Participants describe what they see in the student work. 

Focus Questions: 

What evidence of standards-based work can be found in this student 
product? or 

How does this student’s work indicate performance toward the NYS learning 
standards? 

Sub-questions to guide the discussion: 

Which reading, writing, listening, and/or speaking skills do see exhibited in this 
work?: 

What evidence of real-world, problem-based learning do you find in this work? 

What evidence of technology skills or technology as an instructional tool is 
evident in this work? 

What were the apparent intended outcomes of this student 
assigmnent/project/presentation? 



05’: CLARIFYING QUESTION PHASE 

Participants ask questions for clarification of the presenting teacher. These can questions 
about grade level/content area, or they may be more complex questions about the link 
between what was done in the classroom and what is being presented. 



15’: PRESENTING TEACHER RESPONDS 



Presenting teacher answers questions raised by participants, talks about the intended 
outcomes, the assignment, in relation to what has actually been produced. 

Focus Question: 

What particular classroom practices or instructional methods have influenced the 
outcome of this student work? 

Makes connections between practice and outcome. Reflects on own learning as a result 
of the collaborative review. 
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10’: SUMMARY AND DEBRIEF 



All participate, facilitator guides the discussion 
Focus Questions: 

What questions about teaching and assessment does looking at this student’s 
work raise for you? 

What questions would you like to pursue further? 



15’: REFLECTIVE WRITING 



Teachers reflect individually, in writing. 

Focus Questions: 

What did you learn from listening to your colleagues that was interesting or 
surprising? 

Which, if any, ideas or strategies would you like to try in your own 
classroom? 
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FIGURE 5: CONFERENCE OBSERVATION FORM 



POST-CRSW CONFERENCE OBSERVATIONS: 
FORMAT FOR PEER COACHES 



Peer Coach Name: 

District/school participating in the CRSW conference 

Number of teachers participating: 

Date of Conference: 

Type of student work being discussed: (eg: 7* grade math assi^iment, 2"** grade SST 
essay) 

1 . Describe any connections between teacher practice and student outcome that surfaced 
during the CRSW (eg: giving directions, using scoring rubrics, involving parents, 
adapting to learning styles) 



2. What specific examples of the indicators (identified August 06 by consortium 
participants) did you observe or hear about in the CRSW conference? (e.g.: Student 
defended answers by explaining the math problem in words. Or ‘Students used computer 
to create a web page about their hamster project.’) 



3. Were there any recurring topical themes that surfaced throughout the CRSW? If so, 
please elaborate. 



CONTINUED... 



4. As we discussed at the November 98 peer coaches meeting, you may find it necessary 
or desirable to make changes to the protocol. To help us improve the process, would you 
please note any modifications that you made to the CRSW? (for example, if you 
changed the language in one of the prompts, please tell us the language that you used) 



5. Additional comments or observations: 



If the presenting teacher gives his/her permission, please attach a sample of the 
anonymous student work discussed at this conference. 



PLEASE RETURN THIS FORM by MAIL, EMAIL, or FAX NOT LATER THAN 
WEDNESDAY, MARCH 3 1 , 1 999. THANK YOU 

Ann Martin (awml@comell.edu), Chari Fuerstenau (clf6@comell.edu), 
Diana Straut (dmstraut@sued.syr.edu 
Programs for Employment and Workplace Systems/NYSSILR 
Cornell University/Ithaca, NY 14853. Fax:(607)255-0574 
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